Fed up with issues/bugs

[Oliver]

I’m amazed you keep expecting people to pay for your plugins that don’t work. I’ve asked clients to front up money for the iCal importer which then didn’t work properly and I had to wait for the Event Aggregator to be released. Install this and I get 800+ duplicated events on the next recurring import. I’m now working for free to resolve these issues with an angry client, all because you can’t test features before you sell/release them. Not impressed.

I’m assuming the duplicate issue with iCal is resolved with the latest release? As I’m now trawling through pages and pages of duplicated events deleting them all and trying again.

[Oliver]

So I just tested this locally and its still importing duplicates for all events. My client has 83 events on an iCal Calendar and every time the scheduled import is set to run daily to check for any new events they have added it adds the same 83 to the Calendar, over and over again each time. Frustrating.

[Oliver]

This reply is private.

[Oliver]

This reply is private.

[Oliver]

This reply is private.

[Brook]

This reply is private.

[Brook]

This reply is private.

[Brook]

Thanks again for sharing the feed. That really helped speed up the investigation.

Your calendar is doing the same thing Outlook does. In technical terms it is generating a new Unique ID (UID) each time the calendar is loaded. In the iCal format UIDs are intended to uniquely identify an event. Two events could share the same date, time, and every other piece of information but the ID should always be unique. When an event is updated, it should keep it’s unique ID, because that ID is the only way to accurately identify any event. But your calendar is generating a new UID for every event each time the page is loaded. It is expressly stating that these are all brand new events and should not update existing ones. So, Event Aggregator does as its told and creates new ones.

What program or script are you using to generate this iCal feed? Is there a new version of it perhaps that fixes this bug?

Cheers!

– Brook

[Oliver]

Hi Brook,

Thanks for getting back to me promptly, and apologies for my rant. After the problems with the iCal importer and then this I was at the end of my tether!. Good to know re: import, I guess my only option is to switch of the re-curring import for now and get in touch with the people who generate the iCal feed on their end.

Cheers

Ollie

[Oliver]

Brook, would the same apply to a manual CSV import, I would end up would duplicate events?

[Brook]

No worries at all Oliver! I’ve been at the end of my tether with software lots of times too.

Brook, would the same apply to a manual CSV import, I would end up would duplicate events?

When a CSV is imported we have to make educated guesses on whether an event is a duplicate. Only when you change the title of an event in the CSV and reimport it will you possibly see a duplicate. It the title stays the same across multiple imports, you should never see duplicates.  So no, I do not think you will experience this problem with CSVs.

Let me know if you have any more questions.

Cheers!

– Brook

[Oliver]

So I received this from the developers who supply our iCal feed…

…………………………………………………………

The convention specifies we must provide each event with a unique ID in the exported file. It recommends the ID be made of the current date and time the export is initiated (not the event date/time), and then the domain.

We follow this format and generate a unique ID like “581a5fd307a23-myclarioncall.co.uk”.

The issue they are having is because they have run a bespoke plugin which checks this ID, but they are trying to compare a field which – as per the spec – is randomly generated each time it’s run, hence they’re able to compare the IDs. I am a bit confused how they have used this previously to pickup changes – do you know if this is this a new feature of theirs or have they been using it for a while?

The issue wouldn’t present with other accounts as it’s the bespoke process they run to get the events into their own calendar that is causing the problem. However it’s worth noting we also provide XML subscription format which they may want to consider – this is the ‘raw data’ format, usually used if manipulating data.

……………………………………………………………………..

Looks like I’m at an impasse with this and the manual CSV import is the only option?

[Brook]

Howdy Oliver,

Thanks for getting back!

I was worried that you might get a response like that. The spec is not overly verbose in this area, and so it can be relatively easy to miss some key words causing implementations like that of your iCal feed. But to be sure that is not a correct implementation of the spec, and unfortunately when someone implements it that way there is nothing we can do to consistently work around it. If there was a work around that did not involve innaccurate guessing, we would love to use that instead and account for even broken implementation of iCal. So yes, for now CSV import would be the only option.

We have been building calendar software for nearly a decade now and are pretty versed on such matters. We would be happy to discuss this further with the authors of that software if it will help you out. You could provide us with their email, or provide them with ours:

If you are interested in hearing a technical response to that developer, here it is:

It is easy to see how you might come to the conclusion that the spec is calling for a generated timestamp, and that thus it would change each time the feed is served. But I would ask that you review the relevant section of the spec. As noted at the outset of it:

Purpose: This property defines the persistent, globally unique identifier for the calendar component.

Of course the nature of persistent IDs is that they do not change. If the ID were altered every single time the iCal file is viewed it would be the exact opposite of persistent.

It is also worth noting the full description text. I will bold a few relevant lines for further discussion:

Description: The “UID” itself MUST be a globally unique identifier. The generator of the identifier MUST guarantee that the identifier is unique. There are several algorithms that can be used to accomplish this. A good method to assure uniqueness is to put the domain name or a domain literal IP address of the host on which the identifier was created on the right-hand side of an “@”, and on the left-hand side, put a combination of the current calendar date and time of day (i.e., formatted in as a DATE-TIME value) along with some other currently unique (perhaps sequential) identifier available on the system (for example, a process id number). Using a DATE-TIME value on the left-hand side and a domain name or domain literal on the right-hand side makes it possible to guarantee uniqueness since no two hosts should be using the same domain name or IP address at the same time. Though other algorithms will work, it is RECOMMENDED that the right-hand side contain some domain identifier (either of the host itself or otherwise) such that the generator of the message identifier can guarantee the uniqueness of the left-hand side within the scope of that domain.

This is the method for correlating scheduling messages with the referenced “VEVENT”, “VTODO”, or “VJOURNAL” calendar component.
The full range of calendar components specified by a recurrence set is referenced by referring to just the “UID” property value corresponding to the calendar component. The “RECURRENCE-ID” property allows the reference to an individual instance within the recurrence set.

This property is an important method for group-scheduling applications to match requests with later replies, modifications, or deletion requests. Calendaring and scheduling applications MUST generate this property in “VEVENT”, “VTODO”, and “VJOURNAL” calendar components to assure interoperability with other group- scheduling applications. This identifier is created by the calendar system that generates an iCalendar object.

Implementations MUST be able to receive and persist values of at least 255 octets for this property, but they MUST NOT truncate values in the middle of a UTF-8 multi-octet sequence.

 

It is clear that you have taken the optional “current calendar date and time of day” recommendation to mean the time the feed was viewed. I can completely understand that. The spec is a little ambiguous here in not specifying whether “current” means the time the event was first added to the feed, or the time the feed was viewed. Fortunately though the context of the rest of the spec helps clarify.

The nature of this property is to be persistent. It would be impossible for it to be persistent if the view time was used. Further the spec clarifies that this field is to be used in matching modification and deletion requests. If the ID is constantly changing, if it even changes once, it can not be used for matching. Indeed if it constantly changes, what would the purpose of this field even be?

Some further reading:

• http://stackoverflow.com/questions/25968483/ics-icalendar-uid-purpose-and-use
• https://en.wikipedia.org/wiki/ICalendar#Events_.28VEVENT.29

Please let me know if you have any questions. Cheers!

– Brook

[Oliver]

Thank Brook, I have forwarded this onto the developers of the system to see what they can do. In the mean time I tried a CSV import and ended up with a ton of duplicate events and events appearing wrong days. I’ve attached a private msg with the CSV below to see if you can spot anything.

[Oliver]

This reply is private.

[Author]

Posts