Multiple duplicate events created by Event Aggregator from Google Calendar

[Rocky Hayes]

Hello TEC,

We have a recurring issue with our events being imported from Google Calendar via the Event Aggregator add-on.

First off, a little history – we installed this plugin and the add-on in mid-August. We setup the import from our google calendar via the aggregator and all was well in the land – for a time.

We’re not sure if trouble arrived with the 4.8.2 release of wordpress that we heard caused a lot of problems for a number of plugins, but it was around this time that we noticed multiple duplicate events appearing in our calendar. We submitted a comment at that time on this ticket – https://theeventscalendar.com/support/forums/topic/duplicate-events-in-google-calendar-import-since-4-5-9/ and tried to follow the instructions there.

To be honest, it didn’t go well. We were unable to truly fix the problem at that time – our temporary solution was to delete the plugin and all event data completely, then run a 1 time import of events and turn off aggregation – as in our tests it continued to created multiple duplicate events.

That held us over for a few weeks until the 18th maintenance release (https://theeventscalendar.com/maintenance-release-week-18-september-2017/). Once we updated to the most recent version of the plugins we again did a complete removal of the plugin and data, and setup a fresh daily import from our calendar – and again it appeared all was well.

We were notified by a user on Sept 27th that multiple duplicate events were again displaying on the website. We again attempted to work with support in this thread – https://theeventscalendar.com/support/forums/topic/events-calendar-events-once-again-are-duplicating/ and felt that we hit a wall.

So we thought a much more detailed approach may help. So that’s the history of the issue. Here’s the most detailed explanation we can provide about what we can observe.
———————————
These are the settings that we have in place for import: http://p.tri.be/XdPLCk/3iOlJS3S – you can see that we’d like to bring in all the events from 2017 and moving forward.

Here’s the first sign of trouble. http://p.tri.be/HwnyuG/19qNAuNr – under our scheduled imports we see that 5,061 events have been imported – that number is waaaaay too high. Most days we have less than 1 event – so we can already tell something is not right. Also, it shows that 49 events were updated since the last import, we don’t add events very often – so that number also feels high.

Here’s a list of our most recent imports – http://p.tri.be/hOs6s9/3M8PQSdY

Those are just hints, where we really see the issues is in the events list view in the admin. In this first view there’s something off in the filters at the top of the page http://p.tri.be/8cKw05/30P9jCgp. Notice how the numbers for “mine” and “published” don’t match up – that doesn’t make sense. They should – we’ve not been added any events manually. Also, we don’t know what the “ignored” tab means, but there are a bunch of events there – http://p.tri.be/aseelU/5rYr0RiN

Now it gets really tricky – not all events are duplicated. If you look here: http://p.tri.be/WdxdUs/2PQUOpEv and http://elriad.org/events/ you’ll see that the most recent and upcoming events are showing at least 3 each. But if we look at the top of our list, which are future events for 2019 – these events do not appear to be duplicated. http://p.tri.be/frIEVf/5MUgMOH2

But then we go down a bit – and we start to see duplicate multiple events, it seems normally to be in pairs of 3’s (but hard to confirm due to the scope of the issue) – http://p.tri.be/nW89z4/49JR1zaz One other thing that we notice with this view that seems consistent, and we marked in in that screenshot, in the pairs of 3’s there’s always one event that does not have a user listed in the author column.

Here’s a link to an event that has been tripled – and this is the one that we don’t see an author assigned to in the admin – http://elriad.org/event/oriental-band-weekly-meeting-153/

Here’s the one above it in the admin: http://elriad.org/event/oriental-band-weekly-meeting-291/
And the one above that: http://elriad.org/event/oriental-band-weekly-meeting-429/

Other details you may need:

• The site is hosted with flywheel
• We’re up-to-date with wordpress and all TEC plugins

ical calendar address: https://calendar.google.com/calendar/ical/elriadshrine%40gmail.com/public/basic.ics

———–
We’ll give you whatever access, answers, or help that we can – but we really need to get this solved, and we admit we’re at the edge of our technical capabilities here.

In order to further facilitate your assistance, we’ll post a private message with the login to the wp-admin – please feel free to do whatever you feel is necessary to get this working. We do have daily backups running on the site – so don’t worry about it! We’ve set you up as an administrator.

We’re also happy to provide access to the Gmail account who owns the calendar if necessary.

Thank you so much for reading this very long request!

[Rocky Hayes]

This reply is private.

[Cliff]

Hi, Rocky.

First off, thanks for your detailed message and sorry for all the trouble you’ve experienced.

I’m unsure how to compare http://p.tri.be/WdxdUs/2PQUOpEv to http://p.tri.be/frIEVf/5MUgMOH2, but one thing we’ve received reports of are events with special characters in them. However, this doesn’t seem applicable because of the duplicate event examples you provided.

This part seems the most promising avenue to explore further:

…it seems normally to be in pairs of 3’s (but hard to confirm due to the scope of the issue) – http://p.tri.be/nW89z4/49JR1zazOne other thing that we notice with this view that seems consistent, and we marked in in that screenshot, in the pairs of 3’s there’s always one event that does not have a user listed in the author column.

Here’s a link to an event that has been tripled – and this is the one that we don’t see an author assigned to in the admin – http://elriad.org/event/oriental-band-weekly-meeting-153/

Here’s the one above it in the admin: http://elriad.org/event/oriental-band-weekly-meeting-291/
And the one above that: http://elriad.org/event/oriental-band-weekly-meeting-429/

1)

The duplicate that doesn’t have an assigned author — http://elriad.org/event/oriental-band-weekly-meeting-153/ — also doesn’t have the same Event Description (post_content) as the other two. This obviously looks like a bug, but…

2)

…why would we duplicate it once to begin with (full duplicate)… and then duplicate it a second time (third instance of same event, missing at least the author and description)?

===

If you edit this event — http://elriad.org/wp-admin/post.php?post=24992&action=edit — and go to the Help menu and then check the box to display the Author Metabox, I see there is an author apparently assigned. You’d have to look in the database to know for sure. Could you please look at this post in the database and report back on both the post author and post content?

[Rocky Hayes]

The vast majority of posts have the POST_AUTHOR column set to “1”. Post 24992 and more than a few others have a “0” set for that column – http://p.tri.be/jawX9n/CDg8Qeik

Here’s a screencast of what I can see in the DB – http://p.tri.be/itqEaM/2bpxsauY

I’m unsure how to compare http://p.tri.be/WdxdUs/2PQUOpEv to http://p.tri.be/frIEVf/5MUgMOH2, but one thing we’ve received reports of are events with special characters in them. However, this doesn’t seem applicable because of the duplicate event examples you provided.

The intention was not to compare one to the other – this was simply to illustrate the behavior occurring in both the live calendar view and in the admin posts list view.

[Cliff]

Rocky, just checking in here. I’m still awaiting some help from developers looking into this for us. Sorry it’s taking a bit longer than expected.

While we’re waiting, are there any other trends you noticed that are continuing to happen?

[Rocky Hayes]

With so many events it’s difficult to sift through them all looking for additional trends. The list we’ve provided is as exhaustive as I believe we’ll be able to provide without further guidance on what to look for. Thanks for continuing to look into the issue, we’re excited to have this resolved.

[Cliff]

Sorry for the delay here, Rocky. Our developers have asked for a database dump if that’s possible. You can send a link in a Private Reply (which cannot be done via email).

[Rocky Hayes]

This reply is private.

[Rocky Hayes]

Hi Cliff, how’s it going?

[Rocky Hayes]

Hi Cliff, it’s been 18 days since our original post and over a week since we heard anything from you – can you give us a status update?

[Cliff]

Hi, Rocky. I am sorry for not being in better communication with this issue.

Our Event Aggregator development team has been either been fully out (inconvenient timing, I know) or heavily loaded (unable to dig into this particular issue that you and 2 others have reported).

I, again, am very sorry for the long delay and hope to have a meaningful answer to you next week.

[Cliff]

Thanks for your patience.

After investigating your database, it looks like your events were imported prior to our releasing a fix for author of zero and empty post_content. Additionally, we added even more robust protection against this happening just now.

We put together a plugin for you to try out in your staging environment (of course, we always recommend restorable database and file backups before modifying anything on your production or staging site): https://cl.ly/3b3p0G0p2F2N

It should find all the duplicates, based on the imported iCal event’s UID. All you do is activate the plugin (again, in your staging/testing site first), and it’ll delete the duplicates in batches (so it doesn’t bring down your site). Then see if its deletions look legit. If yes, do the same to your actual production site.

Then, going forward, Event Aggregator should be able to pick up where you left off — for future events — including filling in any missing information (Event Description or otherwise).

===

An alternative method is to wipe out all events — especially if they were all created via EA import and none were created via manual entry in wp-admin — and then just let EA do its thing going forward, since our current code shouldn’t be creating duplicates (since September).

Please thoroughly review these options and get back to me with any thoughts, questions, or report back how things worked for you.

I’m looking forward to hearing back from you.

[Support Droid]

Hey there! This thread has been pretty quiet for the last three weeks, so we’re going to go ahead and close it to avoid confusion with other topics. If you’re still looking for help with this, please do open a new thread, reference this one and we’d be more than happy to continue the conversation over there.

Thanks so much!
The Events Calendar Support Team

[Author]

Posts