[Mark Evilsizor]

Hello Cliff,

It is cool to see that you have access to that level of detail and the aggregator server. But based on your follow up question, I feel like I have not described my process accurately yet. Following is another attempt

* Our site is a WordPress Multi Site Network

* http://www.lindahall.orghttp://www.lindahall.org – this is a publicly available site and it is the source of our events, I believe your servers are able to ingest events from this site as your animated GIF has shown
* http://www.lindahalllibraryfoundation.orghttp://www.lindahalllibraryfoundation.org – this is a publicly available site, I have a scheduled import job which runs on this site and it appears to be working properly
* Inthestacks.lindahall.org – this is our intranet which is only available inside the building, this is the site which is not ingesting properly
* On this site, which tries to import from http://www.lindahall.orghttp://www.lindahall.org, it is only ingesting 3 events and not the 10 it should be
* If I run it live, it brings in all 10, but if I run it from the schedule, it only brings in 3

As part of replying to this email, and due to the earlier trouble we had setting up the TEC Events Aggregator, I decided to wipe the slate clean and try again for this site. I deleted all events (live, trash and ignore) and ran the aggregator, it only found 3 events. I deleted all the events again, and I deleted the scheduled job. Now I created a new scheduled job, the initial run found 10, which it always does when running live. I deleted all the events again (live, trash and ignore), then ran the scheduled job, and it worked as it should, finding all 10 events.

So it seems like all may be well and I was experiencing residue from the time when it was completely broken. I have entered a test event in our source site, http://www.lindahall.orghttp://www.lindahall.org. If that works then all is well and we can close the ticket. I will check tomorrow.

[Mark Evilsizor]

Thanks Barry,

We are currently on Divi 3.11.1

[Mark Evilsizor]

Hello Barry,

Thankfully I do have a clone of this site in our lab network and so I was able to turn off all the network, and site plugins. At this point when I do the test, the bad behavior persists.

When I change the theme to Twenty Fifteen, all the future events show on the first page of the events. In other words it works as it should.

Next I tried changing the theme to match another site on this MU network, http://www.lindahall.org. On http://www.lindahall.org this theme works as it should (or so I though), clicking Next Page properly shows subsequent events. But on the problem site (www.lindahalllibraryfoundation.org), the same theme exhibits the bad behavior.

This led me to wonder about the difference in settings between these two sites
– http://www.lindahall.org allows 5 events per list page and it works great
– http://www.lindahalllibraryfoundation.org allows 10 events per list page and it leaves events in the gap

Experimenting more I found that if I set http://www.lindahalllibraryfoundation.org to allowing 5 events per list page, then it works proper, It shows 5 events, and clicking Next Events shows the remaining.

On http://www.lindahall.org if I set it to allow 10 events per page, it has the bad behavior also. It’s like the Events list page will only allow 5, but when I click Next Events, it thinks that the previous page displayed 10, so it does not think there are any left to show.

If I set either site to allow 6 events on the page,
– On first visit it displays 1-5
– Clicking Next Events displays 7-8
– Clicking Previous Events displays 1-6 (as it should have the first time)

It is definitely some conflict between the Divi theme and The Events Calendar. An interesting note is that the URL is the same when I first visit the events page (and it only displays 5 when it should display more), and when I click Previous Events and it does display all that it should.

https://www.lindahalllibraryfoundation.org/events/list/?tribe_paged=1&tribe_event_display=list

My workaround is to set the limit at 5 events per page, which is okay. But I would be glad to work with you to determine the cause of the conflict as I would imagine many others use this theme with TEC.

[Mark Evilsizor]

So to summarize, here are the open issues we are experiencing with TEC Aggregator

1.) As described in the 8/6/2018 posting above, imported events are not showing up properly in the public facing events list web page. A back and forth is necessary to see these events.

2.) On our Intranet site, which is only accessible inside our organization, when scheduled import jobs run, new events on the source site are not being ingested. But when I run it manually, new events are ingested.
– Regarding this problem, can you elaborate on how the data flows when the import job runs? My concept of it is likely incomplete or flawed, but following is my understanding
a.) My import job contacts a TEC aggregator website with information about the content it wants to ingest
b.) The TEC website accepts the web address, in this case https://www.lindahall.org/events, and checks that site for new or updated events
c.) My import job waits for the TEC website to do step b, then it asks the TEC website for the event information which the TEC website just gathered
– I wonder if the fact that my Intranet is hidden from the TEC aggregator website is causing a problem? Is there a way to look at a detailed log of what the TEC aggregator import job is doing, that would be very helpful in determining what problem this job is running into.

Mark

[Mark Evilsizor]

Sky I am not sure if this is related or if I should open a new ticket, but these ingested events have an odd behavior. Try the following.

1.) Browse to: https://www.lindahalllibraryfoundation.org/events/
2.) Note that there are the following events on the list
August: 3
September: 2
3.) Now click Next Events at the bottom of the page and you see a page that says “There were no results found.”
4.) Now click Previous Events
5.) Note that there are the following events on the list
August: 3
September: 2
October: 1
November: 2

Why didn’t those October and November events show the first time in step 1, or when Next Events was clicked in step 3?

[Mark Evilsizor]

Thank you Victor, I created an event in my source site and dated it 9/1/1967. Then I ran a scheduled import and indeed it did not import this historical event. So that is good!

I tested adding a future event and tried running the scheduled import job on my two consuming sites to see they would bring it in. One site did create the event and one did not. I will wipe this one event out and create another future test event and then take a look on Monday and see if the automatic scheduled import job ingested it over the weekend.

[Mark Evilsizor]

Victor thanks for the updated information. I have installed the latest version and done some tests.

– I am glad you have added improvements to prevent runaway import jobs which I had been experiencing. On my sites we eliminated the condition that triggered this, so I am unable to test that condition again on my sites.

– I did test the issues regarding scheduled jobs and those problems still exist
a.) Past as well as upcoming events are imported on scheduled runs
b.) Scheduled runs are not importing new events from the source site that are in the 3 month date range

Mark

[Mark Evilsizor]

Following up, if I do a one-time import, then the 9/13/2018 event is imported properly.
Mark

[Mark Evilsizor]

Sky the daily import job is running, and it is set to have a 3 month window so that each day it will import any events within the next 3 months from the source site.

From the source site (https://www.lindahall.org/events/ ) you can see that there is a September 13th event. However it has not been imported into the client site (https://www.lindahalllibraryfoundation.org/events/ ) even though the job is running without error.

Can you see if your client tied to my site is doing the same thing?
I am wondering if it is calculating 3 months from the first import date rather than 3 months from today’s date? Or perhaps this is related to the issue of the scheduled runs picking up all the historical events?

Thanks,
Mark

[Mark Evilsizor]

On my test server, there is no routine activity, so the WP Cron does not run. I just logged into it today which triggered the TEC Aggregator to run. It picked up the past events (177).

How did yours run?

[Mark Evilsizor]

Update, I deleted all the events and ran a preview on the existing scheduled event and indeed it wants to get 177 events of which only 3 are after the date set. The attached image illustrates the date parameter, and the past date of the events it wants to bring in.

I clicked cancel from the preview window, deleted the scheduled import, and created a new scheduled import. Now the preview shows 3, and it properly ingested them.

So for some reason, after a few days it wants to bring in historical events rather than respecting the date limit.

Mark

[Mark Evilsizor]

This reply is private.

[Mark Evilsizor]

I did a bit of research on whether or not current recommendations are to block blank agent HTTP requests and it looks like this is still a best practice. We have run the site for years and not had a problem with this rule, so I am reluctant to remove a layer of protection because of all the other bad guys it would allow to interact with my site.

Seems like it should be about 1-2 lines of code to put an agent description in the http request for the media.

So I probably can’t use this TEC purchase until this fix makes it in.

Mark

[Mark Evilsizor]

I was preparing to make a screencast of the sites I used to review the httpd response headers to try and get at the details of this issue, when think I found the issue. I tried some new tester websites and I found one that gave me the 403 that you experienced. Then by reviewing the Apache log I see what the problem is. The user-agent in the request from the aggregator is blank. And at least one layer of our website security blocks requests when the agent is blank.

Two attached images illustrate. The Apache log image shows my request getting the image, and the Mozilla… user agent, and then the request from the test site which has a blank agent and it is blocked.

The curl image shows the default curl getting the file, then when I force the user agent to blank, the request is denied.

It appears that blank user agent is a sign of bad guys on the internet. Can you set your aggregator tool to have a legitimate value for its user agent in the request?

Mark

[Mark Evilsizor]

I would expect you to get 301 status when you try to access http as we are issuing redirects to https.

Our certificates are definitely not self signed, this has been validated by Chrome and other browsers as well as third party checking sites. So for that case I am wondering if the firewall between the box you did the CURL on and our site is setup to do deep packet inspection which would kind of be a man in the middle for the connection and could be using self signed certificate.

I don’t have an explanation for the 403, if you are using a website to test our header and see this, I would love to have the link so that I can look into it.

I did try 3 events from your test site https://wpshindig.com/events/ and the media were ingested properly.

I would be happy to run any debugging tool/setting from your ingest product if that would help your developers understand what it is about our media that is not handled by the TEC ingestion tool.

Mark

[Author]

Posts