Dev.to Auto Publication (10 Part Series)
These next changes came with some architecture changes, which were too intrusive.
- private Paths articlePaths(string folder, ArticleMeta am) + private const(Paths) articlePaths(string folder, const ArticleMeta am)
I marked method parameters with
const and in the case of this method, my unittesting was triggering a warning because the call would produce no side effects.
auto suppressWarning = articlePaths("someLocation", meta);
By assigning to a variable the compiler was satisfied that I was planning to do something with the result.
Thinking more on this for the article, I should have returned immutable as the type is unique and now
const, effectively making it immutable anyway.
I've also implemented the initial interface for making the Dev api into a range. The first part was to remove the dependency of making a web request.
private const(ArticleMe) function(uint page) @safe articlePage;
I did this by using a function pointer which provides a list based on a page number. I can now use this for testing my range without the web request.
Originally I was considering the use of std.concurrency which would allow me to delegate to a different thread, but ultimately did not feel like threading was of benefit over the function pointer.
Inside my function I create a local variable which will hold the web request results. This is necessary because vibe.d use a delegate for its own processing, it is also why I thought the message passing would be helpful.
const(ArticleMe) ret; requestHTTP(format!"https://dev.to/api/articles/me/all?page=%s"(page),
One thing I don't know is how many pages, guess 404 is how I will know to stop.
One thing I needed to change was to modify the save article to operate on a single element instead of a range. This wasn't hard because I already had implemented as a nested function. But I did forget to grab the
This all lead to:
alias inPreviousPull = (x) => !x.isNewerArticle(ps); auto newestArt = devtoMyArticles() .until!inPreviousPull .structureArticles .tee!(x => x.saveArticle(savePath)) .reduceToNewest;
Let's compare with my previous post guess at the implication.
devreqArticleRange .until!inPreviousPull .structureArticles .saveArticles(savePath);
The primary change here is the driving operation is no longer saving the articles. Instead I would like to get the article which is the last one written so I know when to short circuit the next time. This is important because the api will produce unpublished first so I can't just take the first article. With that said I should modify
isNewerArticle to ignore unpublished articles.
tee allows splitting the pipeline execution, this allows me to save each article while I'm finding the newest one.
Generally walking a range multiple times isn't a big deal. Large ranges or in this case where a range would have a larger wait on devices. Though I probably could have just as easily downloaded all the articles into an array, it seems unlikely that even 200 pages would have memory issues in today's computers.
I had chosen to start with pulling the articles to become a little familiar with the api and what my article structure might look like.
Clearly I've gone a little overboard. I could have exported the articles long ago. Instead I'm trying to be more robust and apply applicable attributes.
I can't say that my code was improved because of it though. The annotations were added only as a description of how the function already behaved. That being said, I think my coding style has been heavily influenced by these attributes as I have strived to be able to use them.
I did not write tests first, but obviously have some items to add coverage for.