Hobby Project: Moving to a Range of Articles

These next changes came with some architecture changes, which were too intrusive.

https://gitlab.com/jessephillips/devarticlator/-/compare/v0.0.7...v0.0.8

- private Paths articlePaths(string folder, ArticleMeta am)
+ private const(Paths) articlePaths(string folder, const ArticleMeta am)

I marked method parameters with const and in the case of this method, my unittesting was triggering a warning because the call would produce no side effects.

auto suppressWarning = articlePaths("someLocation", meta);

By assigning to a variable the compiler was satisfied that I was planning to do something with the result.

Thinking more on this for the article, I should have returned immutable as the type is unique and now const, effectively making it immutable anyway.

https://gitlab.com/jessephillips/devarticlator/-/blob/v0.0.8/source/devtopull.d#L78

I've also implemented the initial interface for making the Dev api into a range. The first part was to remove the dependency of making a web request.

private const(ArticleMe)[] function(uint page) @safe articlePage;

I did this by using a function pointer which provides a list based on a page number. I can now use this for testing my range without the web request.

Originally I was considering the use of std.concurrency which would allow me to delegate to a different thread, but ultimately did not feel like threading was of benefit over the function pointer.

Inside my function I create a local variable which will hold the web request results. This is necessary because vibe.d use a delegate for its own processing, it is also why I thought the message passing would be helpful.

const(ArticleMe)[] ret;

    requestHTTP(format!"https://dev.to/api/articles/me/all?page=%s"(page),

One thing I don't know is how many pages, guess 404 is how I will know to stop.

https://gitlab.com/jessephillips/devarticlator/commit/9506d04668a49d33afe37ef252018670dcb6a1bc

One thing I needed to change was to modify the save article to operate on a single element instead of a range. This wasn't hard because I already had implemented as a nested function. But I did forget to grab the in contract.

This all lead to:

alias inPreviousPull = (x) => !x.isNewerArticle(ps);

auto newestArt = devtoMyArticles()
        .until!inPreviousPull
        .structureArticles
        .tee!(x => x.saveArticle(savePath))
        .reduceToNewest;

Let's compare with my previous post guess at the implication.

devreqArticleRange
  .until!inPreviousPull
  .structureArticles
  .saveArticles(savePath);

The primary change here is the driving operation is no longer saving the articles. Instead I would like to get the article which is the last one written so I know when to short circuit the next time. This is important because the api will produce unpublished first so I can't just take the first article. With that said I should modify isNewerArticle to ignore unpublished articles.

tee allows splitting the pipeline execution, this allows me to save each article while I'm finding the newest one.

Generally walking a range multiple times isn't a big deal. Large ranges or in this case where a range would have a larger wait on devices. Though I probably could have just as easily downloaded all the articles into an array, it seems unlikely that even 200 pages would have memory issues in today's computers.

I had chosen to start with pulling the articles to become a little familiar with the api and what my article structure might look like.

Clearly I've gone a little overboard. I could have exported the articles long ago. Instead I'm trying to be more robust and apply applicable attributes.

I can't say that my code was improved because of it though. The annotations were added only as a description of how the function already behaved. That being said, I think my coding style has been heavily influenced by these attributes as I have strived to be able to use them.

I did not write tests first, but obviously have some items to add coverage for.