Google updates GA4 data for a few days after it is initially gathered, so a lot of the methods that we might ordinarily use to add only new data to a BigQuery table won’t work. Much of the introductory material available correctly guides us to just overwrite the entire output table when scheduling queries.

This is the easiest way to get accurate data, and back when we’d just connected GA4 to BigQuery, the costs of doing it for even pretty busy websites were negligible. But as time goes on, and more data is gathered, the inefficiency of querying old data that hasn’t changed has increasing costs. Here’s a better way that isn’t too complicated

What You’ll Need

We’re assuming that you have scheduled queries you run against the whole date range of your GA4 BigQuery export, and that these queries overwrite the results table when they run.

If you need help with getting your reporting up and running through BigQuery, consider this course on how to Prep GA4 Data in BigQuery for Reporting. Note that it doesn’t just cover the basics – later sections of the course will take you through more sophisticated stuff than we do in this post, but they build up to it.

The Plan: Delete and Insert the Last 7 Days of Data, Every Day

Note that there are lots of ways to do this, and some are more robust and sophisticated, but we are going to take the easy route wherever possible. Instead of trying to append new rows, update changed rows, and delete removed rows, we’re just going to delete the last seven days of data from our destination table, then query the last seven days from the export into our destination table.

This might seem heavy handed, but if you are coming from overwriting the entire date range, it’s a much lighter approach.

We’re also going to add a bit of a safety valve. Sometimes Google Analytics bugs out and data has to be backfilled, so we will still overwrite the whole table occasionally. It will also clear up some minor inconsistencies that might arise in your data. This clever idea came from June Li at ClickInsight. We’ve got things set up to do this quarterly in the example below, but you’ll see how you can modify that.

Copy Your Current Tables

We’re going to start by creating test versions of the tables you want to update. The easiest way I found to do this was to take the query I currently use to generate the table, wrap it in parentheses, and stick a CREATE command in front of it.

CREATE TABLE
destination_dataset.destination_table_name
PARTITION BY
Date
AS (
SELECT
cast(event_date as DATE format 'YYYYMMDD') as Date,
-- ...
FROM `your_project.analytics_XXXXXXXXX.events_*`
WHERE
_table_suffix between
-- your first day of data as a YYYYMMDD string, eg 20240101
'your_first_day_of_data_as_YYYYMMDD' and
format_date('%Y%m%d',date_sub(current_date(), interval 1 day))
)

Give your tables a clear name, but note that if you keep them in the same dataset as your production tables, you can just rename them when you are done testing and ready to switch over.

Next, instead of configuring partitioning as part of the scheduled query UI , we are going to define our partitioning in the query itself.

Then comes AS ( the original query ). Obviously, the query needs to include the partition column (which you can name however you please, but ours is just called Date).

Beyond that, you can go to town!

Wait, Should I Test This First?

I highly recommend testing the new versions of your queries against the old ones, because, well, you should always check data for accuracy when you change how it’s processed. In my testing I did detect some minor inconsistencies in one of my queries. In a query to generate a pageview table with session level attribution, I noticed a very small number of pageviews got attributed differently.

I’m guessing these were part of sessions that spanned midnight of the cutoff point seven days ago, and the original attribution data from the beginning of the session was lost. In the case of this query and the data it was running against, the difference was negligible, but that might not always be the case. And, we will run the full overwrite of the tables quarterly, which will keep these inconsistencies from accumulating.

What’s the Easiest Way to Test This?

Here’s the testing plan I used:

1. Create copies of scheduled query result tables (done above)
2. Check table row counts to make sure they are identical across test copies and production originals
3. Create scheduled queries to run on the last seven days for a day or two, do a full refresh, then run on the last seven days for another day or two (we go into this in detail later).
4. You can check your results at any point in the process, but waiting through a couple normal seven day pulls, then a full overwrite, and another couple seven day pulls, reproduces both date ranges for the query, in their natural sequence.
5. Check tables for row level discrepancies using the query shared here by Mark Scannell https://medium.com/google-cloud/bigquery-table-comparison-cea802a3c64d

You can also connect the tables to copies of reports you currently run from your full overwrite tables and compare the final output. This is particularly useful if you turned up row level discrepancies whose impact you need to investigate.

Prepare a DELETE and INSERT Query

Much like we just added some stuff to the beginning of the query we used to overwrite tables to make it create a new table with the same data, we will add commands to the beginning of our old queries in this step. Note that, unlike before, we don’t need to wrap our original query in parentheses. I start with images as they are easier to decipher, what with the colours and indents, but the code is available at the end of this section in copy and paste/screen reader friendly text.

Let’s break this into parts:

We start by declaring and setting some useful default values in variables

• dayof: this is an integer that tells us how many days into the current year we are. The testing version gives the day of the week instead.
• start_date: the date we want to start pulling data from, which defaults to seven days ago.
• start_date_string: same as start_date but in a string data type so we can easily pass it as a table suffix later.

Next, we decide if we are going to do a full overwrite or stick to a seven day overwrite

We use the MOD function with the dayof variable and the number of days we want to go between full overwrites. If the day of the year divided by 91 has no remainder (so, every quarter), we do a full overwrite. You can adjust the number you divide by to set the frequency with which you want to do the full overwrite.

The testing version of the MOD function divides by seven, and the test version of the dayof variable set in the previous step gives the day of the week, starting from Sunday. So in test mode, you’d get your full refresh every Saturday. Perfect for setting this up one week and testing the next, but again, you can adjust the numbers to your liking.

If our modulus is 0, we change the values of start_date and start_date_string to represent the first day of data we gathered. Otherwise, we leave them at at 7 day lookback window we set above.

DELETE the Old

Note that the WHERE clause is using the Date column we set as our partition. Again, you can call this column whatever you’d like, but deleting (and inserting) based on the partition column uses far less data than otherwise.

INSERT the New

Finally, we insert the data from the query we’ve always used into the destination table. The big difference is, instead of always going all the way back to our first day of data, we set the beginning _table_suffix to start_date_string. And most of the time, that start_date_string will only be going back a week – think of the query data you’ll save!

Schedule the DELETE and INSERT Query

There are a couple important differences in how we schedule this kind of query where we specify destinations directly in the query. As you might have guessed, you DO NOT tick the Set a destination table for query results box, as we’ve established the destination in the query and the overwrite vs append behavior in the query itself.

Otherwise this is no different from scheduling a normal query.

I hope you found this helpful and that you’re looking forward to saving a bunch of data. If you have any questions or other feedback, please leave a comment or contact me!

Here’s the plain text version:

DECLARE dayof INT64 DEFAULT (EXTRACT(DAYOFYEAR FROM CURRENT_DATE()));
-- for testing use: DECLARE dayof INT64 DEFAULT (EXTRACT(DAYOFWEEK FROM CURRENT_DATE()));
DECLARE start_date DATE DEFAULT DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY);
DECLARE start_date_string STRING DEFAULT FORMAT_DATE('%Y%M%D',DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY));

IF MOD(dayof, 91) = 0
-- for testing use: IF MOD(dayof, 7) = 0
THEN SET (start_date,start_date_string) = 
(CAST('your_first_day_of_data_as_YYYYMMDD' AS DATE FORMAT 'YYYYMMDD'),'your_first_day_of_data_as_YYYYMMDD');
END IF;

DELETE FROM `your_project.destination_dataset.destination_table`
WHERE Date BETWEEN start_date AND DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY);


INSERT INTO `your_project.destination_dataset.destination_table`
SELECT
CAST(event_date as DATE format 'YYYYMMDD') as Date,
-- ... rest of your query
FROM `your_project.analytics_XXXXXXXXX.events_*`
WHERE

_table_suffix between 
start_date_string and 
FORMAT_DATE('%Y%M%D',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))

The post Slash Your GA4 BigQuery Bills By Only Overwriting Recent Data appeared first on Informed Iteration.

Not only would it be too much to learn at once, but given how recently many people had moved to GA4, there wouldn’t be tons of data gathered yet, so putting effort into controlling the size of queries wasn’t a priority.

Returning to the present, you might have noticed your bills creeping up, or maybe you are just looking to make sure that doesn’t happen. You might just be setting something up, and aren’t sure of what it will cost once all the users hit your reports, so you figure it’s better to build things to be efficient just in case.

Wherever you are in the process, partitioning is perhaps the easiest way to avoid unnecessary BigQuery costs, especially when connecting to Looker Studio.

What You Need to Get Started

We’re going to assume that you already have source tables you make by querying the Google Analytics 4 BigQuery export data, and saving the results manually or by scheduled queries. Figuring out what tables you need is a bit of an art, and figuring out how to make the queries for them is more of something you’d learn in a course than from a blog post I’m trying to keep brief.

For now, I’ll just say you can get a lot done with a session-scoped table, a pageview table, and a leads/sales table. I might write more about how to build those in the future, but if you want something more concrete, I’d recommend this course on how to prep GA4 data in BigQuery.

Another note is that we’ll be setting up new scheduled queries in the examples, but you could just as easily add the partitions to existing scheduled queries if you preferred.

The event_date Column Is Not a Date

The good news is that the raw GA4 export is already partitioned by way of table suffixes. The bad news is that this doesn’t get inherited automatically by tables queried from that raw export. But, setting up your BQ tables with partitions, and getting Looker Studio to take advantage of that, is easy.

The first thing we need to do is establish what column we’ll be using to do the partitioning on, and event_date seems like a logical choice. There’s one small issue, which is that event_date is considered a string in the schema.

So instead of just selecting event_date, we’ll need to cast it as a date in the SELECT part of our query:

CAST(event_date AS DATE FORMAT 'YYYYMMDD') AS Event_Date

You can easily edit your queries to do this where you currently select the date. We don’t have to call the output Event_Date, but keep track of what you do call it as you’ll need it for the next step.

Tell Big Query to Partition the Resulting Table

Now that we have a suitable candidate, we need to tell BigQuery to use it as our partitioning column. There are lots of ways to do this, but I’m going to lean towards a very common and easy one: configuring it in a scheduled query.

When your query is ready, click SCHEDULE above the main editor window. In the sidebar that appears, name and schedule your query, then scroll down to the Destination for Query Results section. Then:

1. Check the box to Set a destination table for query results
2. Choose a dataset and name your new table
3. Put Event_Date (or whatever you called it) into the Destination table partitioning field

Continue through the form and complete the remaining options. Because of the way the GA4 export works, unless you know what you are doing, it is usually best to select Overwrite table (4. above). The other options can all be left as is if you don’t know what they do. Click save when you are done.

Once your scheduled query has run, you’ll have a partitioned table you can connect to Looker Studio.

One More Crucial Click

The last step is to make sure Looker Studio takes advantage of your table partitioning. Whether you are setting up a new data source or reconfiguring an existing one, the process is identical once you get to the step where you edit the connection, pictured below.

Choose the project, dataset, and table as appropriate. If everything went well in the steps above we’ll get a checkbox (1) to Use Event_Date (or whatever you called the column you are partitioning on) as date range dimension. Make sure you remember to tick this, or the queries Looker Studio runs to feed the reports won’t take advantage of the partitioning. Hit the Connect or Reconnect button at the top right when you are done (2).

Thanks for reading, and let please me know if you have any thoughts or questions!

The post Partition Tables for Lower GA4 BigQuery and Looker Studio Costs appeared first on Informed Iteration.

Google WMT Account Creation

We used a dedicated Google Account to set up Analytics to a make sure we could later control and transfer access to the data and settings. Because Webmaster Tools stores similar data and settings, and the goal is to integrate the two tools, we’re going to use the same email we made for GA to set up WMT. Make sure you are logged in under it before going to, or log in under it when asked by, the WMT homepage www.google.com/webmasters/tools/

The first step is pretty straight-forward: enter the website’s address. The only thing to note here is that Webmaster Tools thinks that the www and non-www versions of your site are two totally different things, so type in the one that you usually go by and click the button to continue. Ideally, you have a redirect on your website so that people can only get to one version, and you can use that one. We can use WMT to tell Google which one to use as well, but we’ll leave that for another day.

The next screen will offer a Recommended Method tab, and Alternate Methods tab. The recommended method will change based on your situation: if you already have GA set up and the Tracking Code on your site, then verifying through Google Analytics will be the recommended method. If it isn’t, just click the alternate methods tab and select the Use Google Analytics button. Whichever tab you are on, click the Verify button when ready.

You should be on the Congratulations screen with a green checkmark. GA and WMT both need you to put something on your site, but since we can use the GA one to make WMT work, but the WMT code won’t make GA work, it makes more sense to set up Analytics first. Then you only have to add one thing to your site.

But we aren’t quite done. I almost forgot this myself (seriously I’m doing this along with you to make sure I don’t miss anything and get screen captures).  There’s one more thing to do before we log out of our data only email account. Click where it says Webmaster Tools in red (yes it’s a link) twice, once to get to the home page for the given website, and again to get to the page shown below.

Adding Users to Webmaster Tools

From the WMT website selection page, click the Mange Site button to the right of the site we’re adding a user to to get the dropdown menu shown in the green square on the picture above. Select Add or remove users. On the next screen, click the red Add a New User button at the top right, or the blue Manage Site Owners link just to the left of it.

WMT permissions come in three flavors: Restricted Users can just look around; Full Users can change a few things about how Google sees your website (enough to do some damage), and Owners can do everything, including locking out other Owners and telling Google that you don’t want your site to show up in results anymore. Be careful who you give Full or Owner access to, and if they stop working for/with you, remove them from WMT promptly. Remove them immediately if it was an acrimonious split.

The Add a New User button will bring up a pretty self-explanatory pop-up window. Enter an email, select the permission level, and click Add. The Manage Site Owners link takes you to another screen. Click the Add an Owner button at the bottom, add an email, and click continue.

Integrating WMT and GA

A lot of the data that WMT gathers is actually a lot easier and more useful to look at in Google Analytics, so you should always link them together. And it’s just a few clicks. Click the red Webmaster Tools text link at the top left to get the website selection page, and hit the Manage Site button again. This time, select Google Analytics Property. Click the radio button for the GA Web Property that gathers the data for the website we’re managing, then the Save button. Note that you can only have a website associated to one property, and doing this will cancel any previous links to other web properties in your Google Analytics account.

There’s a lot more that you could, and probably will, do with Google Analytics and Webmaster Tools, but all of that could arguably be called configuration or even customization. As far as basic setup, you’re done. Congratulations!

We’ll eventually look at good first steps to take now that things are set up, like that www vs non-www thing we mentioned above, and setting up conversions on your contact pages. For now, I’m ready for a break. Please leave a comment if you have any questions.

The post Google Webmaster Tools Setup and Analytics Integration appeared first on Informed Iteration.

This will save you having to log in or out of your main account when you want to look at your analytics. If you have several websites, or manage the sites of several clients, this is a lifesaver. It’s also the procedure you use to give consultants, clients, employees, and so on, access to your account.

Adding Users to Google Analytics

If you’ve come right from the last post, click the account name at the top of the page, just under the orange menu bar, directly to the right of where it says Account List. If not, click the home button at the top left, then the Admin at the top right, then the name of the account. Once you’re on the screen pictured above, click the Users tab I’ve highlighted in green. The Users Tab will have a New User button, just below the tab itself. Now click it to get to the screen shown below.

This is pretty straightforward: enter the Google registered email you want to give access to, decide if they need Admin or User privileges, and if they need to be notified by email. If you are giving someone user privileges, you can restrict which profiles they have access to. When you click the Admin radio button, the profile selection area disappears. You don’t have that control with Admins because Admins can see all the profiles in an account. That is why we go to the trouble of making a separate account for each business.

Don’t forget to click Add User at the bottom when you are done. Eventually you get in the habit of scrolling down to look for these buttons any time you do things in tools made by Google. You can make sure everything worked by checking under the Users tab for the emails of the people you are adding.

Adding a Master Profile

Google Analytics (GA for short) comes set up with one profile that collects all the data it can. Eventually, you or another user will want to filter the data. Filters are one of the most powerful features in GA, and they are used to modify data in all kind of situations. But like anything that is powerful, filters can get pretty complicated, and the outcomes of using them aren’t always what one anticipated. Beyond this, data that gets filtered out can never be recovered.

As in many other technology applications, the smart thing to do is have a back-up: a set of data that has everything, and that no one ever puts a filter on. Ever. And if you do it while you are already logged in it could take under a minute. Click the home button at the left of the orange menu bar, then the Admin button at the right of said bar. Now click the account name we want to work on from the list, then the property name from the next list. You should be in the Profiles tab for the website we want to make a back-up for. Click the New Profile button.

This is again, pretty straightforward (it’s not unusual that GA makes something harder to find than it is to actually do once you find it). Profile names like Master, Back up, All Data, or so on, will be immediately recognizable to frequent Analytics users, who will know not to filter them. We can guard a bit against someone given Admin access getting a bit too ambitious by explicitly including Do Not Filter in the profile name. I go one step further and prefix the name with a “z” so that it will show up last in lists.

You’re Done! Kind of…

There are still a couple or three things that you’ll have to do before you’ve done good job of basic Google Analytics setup. Most importantly, you need to get the tracking code up on your site. Then you can integrate your GA account with Google Webmaster Tools and get information about your how people find your site in Google search.

The post First Steps in Google Analytics Configuration appeared first on Informed Iteration.