Author Archives:

Home / Articles posted by

Scraping websites

I am currently working on the migration of a website. The old one – after ten years – has a lot of content, including more than 3800 documents in different shapes and sizes. To complicate matters, they are hosted on a web server that is not directly accessible, and in a CMS that is not very user-friendly.

Scraping the html content was relatively easy. It allowed me to make a sitemap and find all active pages, the resources linked to them, download their content, and the date they were last modified. Finding the same for the PDFs and DOCs was harder to do.

A preview with metadata was available, yet the extension was .pdf or .doc, and the scrapers I used do not follow those links.

The solution was to get into the javascript of, and there remove the filter to exclude documents. And voila: the crawler follows the links and captures the content of the preview.

It allows me to see where pages link to documents, which documents just sit there, and when they were uploaded or last modified. Whilst you still have to go through some of the records manually , you can now decide on 95% of your content when you implement simple rules, e.g.:

  1. Delete stubs and content older than 5 years
  2. Delete all vacancy announcements older than 30 days
  3. Keep all annual and financial reports, etc.

Should you migrate your website, then scraping may be an option. It may also be useful when you want to do a spring clean. But better still: get a decent CMS and make sure that it is managed!

Giving meaning

Ok communications people,

When asked, what communications actually is, one of the preferred answers is ‘giving and sharing meaning’. You are wrong.

Because people who give that answer are working for corporations, governments and NGOs. Giving and sharing meaning only have their place in the lifeworld. Once you are institutionalised, i.e. sell your soul to the corporate or non-profit devil, you are part of the system. Read Parsons, Luhmann, and Habermas to get my point.

The system – you are part of – has a formalised mandate, based on the public discourse of a time gone by, and you can only share meaning if you are willing to debate that mandate. If you don’t, and most of my colleagues do not, then you are an instrument of the status quo, your master’s voice and part of the colonization (Frankfurter Schule) and normalization (Foucault) of society.

Yes, it is that bad. You are pretending to be Robin Hood, but you are his master’s voice. What do I want? Go into that management meeting or the board room, and tell them times have changed. You need to engage on three levels. And not to mitigate or manage risks (Beck), but to change wisely.

Change slowly, or fast. But don’t stay the same. Because then you lose. And everytime you keep up the myth you are sharing meaning, you perpetuate the myth. And you lose time, and we don’t have time. Because change needs to happen now.