Most of my ideas revolved around some kind of communication/training for those stakeholders. This led me to consider: How does someone get better at asking questions, and how does one help others develop their curiosity?

Research So Far

At the time I write this, I’ve read Hal Gregersen’s Questions Are the Answer (QAA) and taken a course in Question Formulation Technique (QFT). Gregersen is a senior lecturer at MIT Sloan; the QFT course was from the Harvard Graduate School of Education. I’m not trying to pretend that this makes me an expert. I’m trying to make the case that I am following the lead of industry-leading experts.

QFT is about getting students to ask more questions by encouraging them to come up with as many questions as possible in a timed setting, then working with the students to see the value of those questions.

QAA covers a lot of ground, exploring how corporate leaders foster curiosity in themselves and by creating environments where their employees can feel not just safe, but encouraged, to ask tough questions. It relies a lot on inspiring anecdotes about famous business leaders. As a result, a lot of the takeaways need to be scaled down for those of us who aren’t the CEO of a massive company. One of the things that doesn’t need much downsizing is what Gregersen calls a Question Burst, which is quite similar to a QFT session.

We’ll look at both of those in more detail later. For now I’d like to focus on what they have in common.

Both QAA and QFT focus not so much on what the individual can do to be more curious, so much as the need for leaders to foster environments where questions are welcome and safe to ask, whether that’s an office or a classroom. It’s hard to motivate someone to ask more or better questions when the situation they are in fails to incentivize that, or actively punishes it.

With Questions, Less is Usually Less

Another point raised in QAA and QFT is that we don’t need to learn how to ask questions: We need to un-learn all the self-censoring we’ve mastered.

As any parent could tell you, children start off good at asking questions. Then our parents teach us, quite correctly, that not all questions are appropriate at all times. Then we go to schools, where we are judged more harshly on our questions. We are also introduced to the concept that there’s a lot of correlation between who asks the questions and who is in authority.

Finally, we end up in workplaces. The wrong question just went from getting you detention to getting you fired.

The result is that children go from asking hundreds of questions a day, to a handful, as they close in on adulthood.

That’s all pretty well researched per QFT and QAA. The following takeaway is not, but I’m still comfortable making the call: the odds of you self-censoring a good question into oblivion are probably a good deal higher than those of you having a stupid question. Unless you are asking questions with the unfiltered cadence of a toddler, you could probably stand to ask more questions.

Because worrying about the appropriateness and consequences of a question are already front and centre for most people, trying to get people to ask “better” questions could have a dramatic chilling effect. Imposing yet another internal checklist to apply to questions before sharing them could easily do more harm than good. The road to better questions goes through the land of asking more questions, seeing the impact the best of those questions have, and letting that influence how both individuals and the organization approach challenges.

For people to ask better questions, they have to start asking questions, and if they are going to feel safe asking meaningful questions, leaders need to make sure that is the environment they are building.

Getting Cozy with Curiosity

As mentioned earlier, a lot of QAA isn’t so much methods to apply as inspiring ideas to emulate. Plenty are good to go as is, but some wouldn’t really fly for people outside the C-suite of a very large company. Hiring a clown (literally) to come to board meetings and act as a court jester, poking fun at the leadership, isn’t exactly something a middle-manager makes happen.

Almost all of them, however, are about getting out of our comfort zone. The more we are exposed to new ideas, the more our mind is able to shift into a curious, exploring mode. That kind of thinking can atrophy the more we develop experience and specialization, making it harder for us to adapt.

Here are the things I am trying to apply, some taken as is, and some with scale adjustments for folks outside the C-Suite:

• Explore the World

This one starts out as “live abroad as much as you can” but that is a big ask. For us non-CEOs, I’m replacing it with travel that allows you to interact with locals in their own environment.

• Expose Yourself to Art

This one starts out as “make a community with live/work spaces for artists to experiment and create in, then immerse yourself in it.” We’re paring it back to “go see something at a gallery or theatre made by someone with a different viewpoint than you.”

• Talk to People Outside Your Normal Circles

This begins as an anecdote about a CEO getting a ride from the airport to the office from a delivery driver instead of waiting a few minutes for a delayed limo. The benefit wasn’t the time saving though: the CEO spent the ride talking to the driver about their job and interactions with the company. I don’t have a network of delivery trucks, but I do take a good deal of rideshares, and always try to converse with the drivers (if it’s obvious they aren’t the chatty type, I let it go).

• Meditate

This one works for Fortune 500 leaders like Marc Benioff all the way through to monks that have sworn vows of poverty. Put your mind in a different state from time to time, daily if you can manage it.

• Do Something You Don’t Already Know How to Do

Forcing yourself to be a beginner at something is a great way to build the perspective that there is always more to learn, that you don’t have all the answers, and that being honest about what you don’t know is often the fastest way to learn.

• Don’t Rush to Solutions

When we haven’t decided on a solution, it can be very uncomfortable. Thinking about a problem feels (and in most orgs, is seen as) a lot less productive than working on a solution. But accepting a solution rescopes the whole mental space we work in. Instead of asking “how do we improve xyz,” we are now asking “how do we apply abc to improve xyz.” Obviously, solutions do need to be found and applied, but try to get comfortable with the headspace that accompanies comparing solutions.

• Don’t Hype Solutions When You Want Honest Feedback

Don’t start a conversation with “I have been working really hard on using abc to fix xyz for several weeks now and I think there’s been a lot of progress” if you want honest feedback. Try “I am thinking about xyz and wondering how you might approach it?” Save the fact you actually have a solution in mind as a follow up question, and downplay your attachment to it.

While that helps me get on the path to asking more questions, these aren’t things that I can impose on my clients (as much as I’d love to take an afternoon off to visit an art gallery with them). Gregersen also shares a method he uses in his professional practice, called a Question Burst.

We’ll look at it in a moment, but I want to switch back to the Question Formulation Technique at this point, as the two are quite similar.

Question Formulation Technique

The most robust methodology I could find to get people asking more questions was QFT. QFT was developed by the Right Question Institute as part of community organizing work, and introduced to educators in the 2011 book Make Just One Change by co-founders Dan Rothstein and Luz Santana. Despite knowing it may have limited applicability in a business setting, I wanted some exposure to it.

It was a great course about a great method. And one with applications well outside the classroom. Most of the people taking the course were educators with access to students: I had to use my girlfriend as my “class” for homework purposes.

Instead of basic geometry or the civil rights movement, we set a four minute timer, and she generated a list of questions about a trip we are planning. While our topic was frivolous compared to my classmates’, we went from our question list to having the whole trip planned in about 90 minutes, which is a record for us.

For any educational context, or personal or community-based endeavours, where a transparent exchange of thought would help, I’d consider rolling out a QFT session. But I wouldn’t use it in a business setting for two reasons. The simple one is the very generous CC licensing only extends to non-commercial use.

Beyond that, though, there’s no next step a client or boss would get excited about. An educator needs to reintegrate this with their lesson plan, answering questions or guiding students to find their own answers. It can get you closer to finding a solution in the sense that exploring a problem helps you solve it, but the goal of the technique is to get better at exploring topics, more than it is to arrive at a solution.

Question Formulation Technique Resources

The Right Question Institute – What is the QFT?

Question Burst

The Question Burst method, as mentioned above, is quite similar to the Question Formulation Technique. Both feature a timed interval where participants generate as many questions as possible, then engage in exercises to refine and prioritize their questions. The nature of those exercises and what happens with the prioritized questions is one area where things differ.

Another is that in QFT questions are based on a prompt, which is a sentence or two, a quote, or even something more abstract, like an image or piece of music. Question Burst instead allocates up to two minutes for the facilitator to provide information on the topic.

QFT is appropriately open-ended so that students can explore new ideas and educators can weave it into their teaching in appropriate ways. Question Burst’s extended context and included follow-up questions provide structure that leads to taking action, and setting a plan to continue using curiosity to move forward on a challenge, in the absence of an instructor.

Question Burst Resources

Question Burst Guide

Question Burst Modifications for Discovery

The Question Burst method as presented in the above guide and described in QAA is an excellent method that I can recommend for any environment where people are already pretty comfortable with asking tough questions. I’ve used it in discovery contexts (for new projects with current clients) with some modifications: Specifically, I take the prioritized questions away, categorize them, and come back with some answers and ideas on how to move forward.

I’ve really enjoyed the results. I’m not going to get into my theories on why discovery can be hard on the people trying to explain what they need. I’ll just say that letting them ask the questions instead of having to answer mine cuts away a lot of the more stressful and performative aspects.

One thing that remains, in the Question Burst context and in pretty much every meeting, is that bosses tend to set the tone to an extent that often impacts the value of the activity negatively. And strangely, it has very little to do with the people in the room. If no one talks before the CEO at any meeting the CEO is in, that establishes the norm. People who have no problem cracking a friendly joke at their boss’s expense, or expressing their disagreement with that boss when necessary, can still be hesitant to speak before that boss does. Particularly in a new situation – like when a consultant is around.

Unattributed Question Bursts

The Question Burst as described in QAA involves participants throwing out questions while someone, usually the facilitator, acts as a scribe. One issue I’ve had with this is the scribe’s typing/writing speed often becoming a bottleneck. The bigger one is the issue above of people wanting to let senior colleagues go first, which anchors the rest of the session. Even where people are equal down to the pixel on the org chart, some people have more confidence or charisma than others. They might ask more questions, or people might be more hesitant to ask questions that run contrary to theirs.

My response was to try running Question Bursts where people write their questions down and prioritize them, then send them to me as private messages. This alleviates what I’ll understate as the “shyness” problems, even in groups small enough to make guessing who asked what somewhat obvious. Unfortunately, it means cross-pollination of ideas is either eliminated, or relies on some lightning-quick cutting and pasting by the facilitator, which becomes impossible for all but the smallest groups.

Hiding attribution, while fostering cross-pollination, and allowing the facilitator to manage the session smoothly, seemed like it might have a technical solution, so I built version 0.1. My next post will explain how it works, and I am planning to run a demo at MeasureCamp Toronto on Oct 3, 2026. Hopefully I will see you there!

The post Ask more questions; get better answers appeared first on Informed Iteration.

There was a lot of info I wasn’t able to squeeze into my talk for the 2019 AB Testing Summit, including proper reference or sourcing. I’ve supplied that info here, along with a few comments I hope will be useful to those looking to do further reading about Bayesian AB Testing in Marketing.

Also, if you have a question that won’t fit in Twitter, please leave a comment and we can have a more detailed discussion that will be easier for people with similar questions in the future to find.

You can find the slides here.

Introduction to Bayesian Methods in Marketing

As a first look at Bayes Theorem, you could do a lot worse than An Intuitive (and Short) Explanation of Bayes’ Theorem

We almost immediately get into considering where Bayesian Priors come from and the validity of these processes, which is hotly debated. By extension, we need to consider several sources to get an idea an informed perspective.

If you don’t read any other source I provide, 5 Reasons to Go Bayesian in AB Testing – Debunked is probably the fastest way to get see a well argued perspective that diverges from the generally accepted hype. If you want to learn more, the author, Georgi Georgiev, is knowledgeable and prolific. There are several other great articles and whitepapers that explain how Bayesian methods may not always live up to the accompanying spin, which were immensely useful in building my understanding and preparing this talk. We’ll mention some more of his work later, but the one blog link above and two following white papers offer a good overview of the contention.

• Issues with Current Bayesian Approaches to A/B Testing in Conversion Rate Optimization
• The Google Optimize Statistical Engine and Approach

The overwhelming majority of discussion about Bayesian methods in AB testing was dismissed as coming from sources with a clear conflict of interest paired with an almost complete lack of transparency, and in some cases, presented with total incoherence. The pro-Bayesian materials we looked at that were useful were from referred journals or technical whitepapers with a high degree of transparency, for which the vendors should be commended.

• Bayesian Statistics and Marketing – Published back in 2003, this lays a lot of the groundwork about how Bayesian methods work and why they are useful to marketers, even though it predates the prevalence of AB testing that online businesses enjoy today.
• Bayesian A/B Testing at VWO
• The New Stats Engine (at Optimizely)
• If you know where I can get my hands on a Google Optimize white paper let me know.

We also looked at a non-Bayesian modern CRO tool that claims to deliver similar advantages of those claimed by Bayesian tools. We wanted to establish if the claimed advantages of Bayesian tools are only possible with Bayesian tools, or if non-Bayesian methods that are appropriately advanced and customized for CRO can deliver them as well. It may not be a huge surprise that this method was created by the aforementioned Georgi Georgiev.

• Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method

Introducing Bayes Visually With Python

Introduction to Bayesian Inference is a great first look at how doing your own Bayesian analysis could work, and even if that isn’t your goal, it provides a very helpful visual interpretation to how Bayes works in marketing.

The 3d graphs were based on, like most of the code holding the internet together, a Stack Overflow answer.

Ease of Explanation

The Part to Ignore

Sorry, but I’m going to include a bit of a rant that I didn’t have time for in the talk. Skip to the TLDR if you’d like.

The whole “frequentist methods are ridiculous because [some variety of absurd claims about how frequentists aren’t allowed to discuss results or use their brains except in the strictest possible interpretation of the math they use]” such as those included by Google in their Optimize documentation are a text book example of fallacious “straw man” arguments.

Simply put, you make up a version of your enemy that doesn’t reflect reality, but completely supports your argument, then argue against that instead of what the actual enemy says.

This argument has been raised and conclusively shut down repeatedly, perhaps most devastatingly  by A “Bayesian Bear” rejoinder practically writes itself… where a mathematician gives a hilarious example of what happens if you make a Bayesian straw man to compare to the Frequentist one made by unscrupulous Bayesians.

TLDR: If you hold mathematicians to the strictest possible requirements of their assumptions and definitions, the only people who will understand are other mathematicians, and even they will think you are a jerk that is just wasting people’s time. For Bayesians to pretend Frequentist methods are the only part of statistics facing this issue reduces the credence we can give other claims made by this particular subset of Bayesian proponents.

The Part That Is Interesting, But Still Kind of Esoteric

Initially I didn’t fully absorb the importance of the debate, but the aforementioned Georgi Georgiev was kind enough to have an email chat with me and I was able to understand more of the difference in positions.

Bayesians assume that their best guess of the probability, is the probability.

Frequentists figure out how unlikely an outcome would be if the experiment were repeated a very large number of times. It’s still up to the user to decide what this result means in their real world situation.

Using a Bayesian tool doesn’t just mean trusting that the priors are chosen in a way that will work with your data, it also means trusting that the method is generating an actual probability. For people who think about these things deeply, that can be a pretty big leap of faith.

To quote Georgi: “Frequentist methods give you data and uncertainty and make it crystal clear that the decision-making part is up to you, and cannot be purely data-driven.”

Now, it’s still up to us as practitioners to decide how to explain what level of detail to stakeholders, and not everyone at a meeting about what creative to go with is going to be up for a talk about p-values. For people who are, though, it actually opens up a fascinating area of discussion and introspection.

We build human assumptions, often based on qualitative inputs, into the tools and methods we use to gather and report on data – but we often consider the results to be purely quantitative.

And we didn’t need to make a big AI project to run into that issue – even a little old t-test is an opportunity to see to what level we separate our assumption and interpretation from the data itself.

Optional Stopping and Bayesian Suitability for CRO

There’s a lot of material out there about optional stopping, but here are some excellent reads on the topic that go deeper than your average blog post without turning into heavy math and coding.

• Bayesian AB Testing is Not Immune to Optional Stopping Issues
• How Not To Run an A/B Test
• Is Bayesian A/B Testing Immune to Peeking? Not Exactly

Faster CRO Results

Information about specific products comes from either the aforementioned whitepapers or website site documentation accessed over the month of May 2019. Note that at some points while speaking (and I think even one slide), I use loss control and error control interchangeably, which is inaccurate. Loss control and error control are very different things.

Recommended Non-Bayesian Products

Optimizely, VWO, and Google Optimize, even once the hype is taken away, all make compelling cases to any org without the data science resources to do their own modelling. There are some non-Bayesian alternatives, however, that allow users to compare Bayesian tool results to traditional or more advanced frequentist menthods, for free or very little investment.

Google Analytics users can take advantage of Stéphane Hamels’ DaVinci Tools. This Chrome extension, even in its free version, makes a million and one quality of life improvements to the Analytics UI, as well as adding some useful features. One of these is the ability to take a t-test by clicking a few times on any Google Analytics report. If you also use Optimize, your experiment data is already in GA in a format that makes this very easy.

For a more sophisticated approach, have a look at the Agile A/B Testing Calculator included with Georgi Georgiev’s Analytics Toolkit. It’s very inexpensive, and Google users can connect it to Analytics/Optimize easily.

Sorry for the Google-centric view but it’s what I spend most of my time on. If there are other tools out there you feel provide a good way to double check what the Bayesian “Big Three” are recommending, I’d love to hear about it.

Please let me know in the comments or on Twitter, or reach out with any other questions or comments!

The post Bayesian AB Testing in Marketing Reading List appeared first on Informed Iteration.

Subsequent posts were delayed by the very severe illness, culminating in the passing, of my father.

Alex Amprimoz’ Development Legacy

(not to be confused with Alex Amprimoz’ Legacy Development)

My dad (pictured at left) was a French professor at Brock University here in Niagara, but loved computers with a rare passion. I grew up in a house full of computers, any one of which would often be spread out in pieces across the dining room table, much to my mother’s chagrin. Dad went as far as picking up a degree in computer science, then doing IT consulting on the side while still teaching French.

During my last conversation with him I made it a point to tell him about my getting involved with Software Niagara because he’d be happy to hear about it: not just my being involved, but that something like that was taking place locally.

Being a Niagara SEO Consultant Can Be a Little Lonely

Software Niagara is a grassroots organization looking to unite and foster the Niagara development community.  Their first event, a Democamp, took place last September, and was absolutely great. Seeing that kind of creativity in my own backyard got me very excited: I work predominantly online and most of my clients and collaborators don’t live nearby. Seeing a chance to help build a tech savvy community locally I decided to volunteer.

Their next event is this evening. I know, it’s short notice. But if you can’t make it tonight, you can come next Monday, or any Monday thereafter. Software Niagara will be doing a Coworking Night every Monday at the Starbucks on Lake St. from 7pm onwards. Grab your laptop and go work at Starbucks, like many of us do anyway. But this way you aren’t doing it alone.

Coworking Niagara

Software Niagara isn’t the only opportunity to cowork in Niagara. There is also the (aptly named) Coworking Niagara group, which meets from noon until four every Wednesday at the Mahtay Café on St. Paul St. Again, grab your laptop and come meet other tech, creative, and entrepreneurial types while you work.

I’ve been meeting with this group for the better part of this year. What started off as a chance to do a bit of networking developed the kind of business friendships that have led to a software joint venture and several collaborative opportunities. Perhaps more importantly, the friends I’ve made there have been tremendously supportive through the loss I’ve recently experienced.

If you work in an office you have that face to face team presence. As an SEO consultant working online I don’t get that with many projects, so it’s great to have a group to be social with while still getting some work done.

Uhh, What’s Coworking?

Oh, sorry. I haven’t really touched on that have I. Here’s the explanation from the Coworking Niagara Event page:

“Coworking is a work structure where freelancers and other independent, creative professionals get together in a shared space for the purpose of being more productive, generating new ideas and solving knowledge work problems.

Coworking is not a networking event, it’s not a support group (although those things happen). It’s a place and a structure to help you get your work done and produce a higher quality output.”

As always, thanks for reading: even more than usual as today’s post had a more personal and less informative flavor. The next post will likely get us back to Google Analytics setup. In the meantime, I’ll see you at a coworking event, unless you’d rather leave a comment or get in touch through social media.

Where do you like to get work done?

The post Loss, Legacy, and Local Software appeared first on Informed Iteration.

There are a stack of people who’ve helped make this possible, and I’ll be thanking as many as possible, hopefully in person, soon. For now, one person stands out, having repeatedly shared invaluable advice as I thought about how to turn being really good at something into a business doing something.

I was lucky enough to know James Chartrand over Twitter, and even more fortunate to start working with her. For those who don’t know, James is the founder of the celebrated Men with Pens web design firm and Damn Fine Words copywriting course. She’s also an awesome small business adviser, and when I give you answers you can understand instead of confusing eight-page reports, you’ll join me in thanking her.

So with the help of her and many others, the site is ready. Note that it is not done. For me to say it was done would contradict the entire point of my business: a business website should never be done.

Succeeding online is about continuously creating content of value to people who’ll be interested in your products, and promoting and presenting that content. A business website isn’t a brochure or billboard, it has the power to come alive, and go beyond interaction to engagement.

In order to practice what I preach, I want to make this blog a place for small business owners and marketers to get advice and exchange ideas about how you connect with customers online. And I want my social media presence to be an extension of that.

So please leave a comment or get in touch though social networks about questions you have and things you’d like to see me deal with here: I want to make this a great resource that you’ll want to engage with, and I need your opinions to do that. Thanks for reading and I look forward to hearing from you!

The post Welcome to the Brand New Website! appeared first on Informed Iteration.