In this article we will explore one of my workflows leveraging ChatGPT capabilities when working on automated software testing.
If you’re interested into ramping up your developer productivity with AI you might be interested in my other article about software architecture visualization using ChatGPT and PUML.
The Iterative Prompting Workflow
The general workflow I’m using consists of the following steps:
Initial prompt with limited information.
Review and refine output.
Prompt with fixed output and instructions to apply further improvements.
ChatGPT generates improved output according to provided fixes.
Repeat the process until the output meets your criteria
What is powerful about this workflow is the ability to provide feedback on specific parts of the generated output, and have the AI model apply improvements to the entire output.
This approach can be used for different applications for example writing or learning new concepts and iteratively asking questions to deepen your understanding.
Unit Testing via Iterative Prompting
Let’s dive into unit testing example. In my example I will ask ChatGPT write unit tests for the PaymentOrderProcessor class. There’s bunch of dependencies in this class and several things going on that can result in successful or failed outcomes. Additionally different types of Exceptions may be thrown during execution.
I’m using PHP and phpunit for testing, however this approach can be applied to any unit testing framework or language.
The PaymentOrderProcessor class definition is as follows. We will write unit tests for its processPayment method.
class PaymentOrderProcessor {
private PaymentOrderRepositoryInterface $repository;
private NotificationSenderInterface $notificationService;
private APIClientInterface $apiClient;
private LoggerInterface $logger;
private FraudDetectorInterface $fraudDetector;
public function __construct(
PaymentOrderRepositoryInterface $repository,
NotificationSenderInterface $notificationService,
APIClientInterface $apiClient,
LoggerInterface $logger,
FraudDetectorInterface $fraudDetector
) {
$this->repository = $repository;
$this->notificationService = $notificationService;
$this->apiClient = $apiClient;
$this->logger = $logger;
$this->fraudDetector = $fraudDetector;
}
public function processPayment(PaymentOrder $paymentOrder): PaymentOrderProcessingResult
{
if ($this->fraudDetector->isSuspicious($paymentOrder)) {
$this->logger->warning("PaymentOrder {$paymentOrder->getId()} marked as suspicious");
$this->notificationService->sendNotification($paymentOrder, false);
return new PaymentOrderProcessingResult(false, "Payment marked as suspicious");
}
$paymentOrderApiResponse = $this->apiClient->sendPaymentOrderRequest($paymentOrder);
if ($paymentOrderApiResponse->isSuccessful() === false) {
throw new PaymentOrderRequestFailure($paymentOrderApiResponse->getMessage());
}
try {
$this->repository->savePaymentOrderResult($paymentOrder, $paymentOrderApiResponse);
} catch (PersistenceException $e) {
$this->logger->critical($e->getMessage());
return new PaymentOrderProcessingResult(false, "Failed to save payment order result");
}
$this->notificationService->sendNotification($paymentOrder, true);
return new PaymentOrderProcessingResult(true);
}
}
Prompt 1:
I requested for a unit tests and provided a definition of class PaymentOrderProcessor
Write unit tests for the following class. Class test mock properties names should end with "mock" postfix {class definition}
Output 1:
Here is the whole body of the test method generated by ChatGPT:
public function testProcessPaymentSuspicious(): void
{
$paymentOrder = new PaymentOrder(1, 100, 'USD');
$this->fraudDetectorMock
->expects($this->once())
->method('isSuspicious')
->with($paymentOrder)
->willReturn(true);
$this->loggerMock
->expects($this->once())
->method('warning')
->with("PaymentOrder 1 marked as suspicious");
$this->notificationServiceMock
->expects($this->once())
->method('sendNotification')
->with($paymentOrder, false);
$processor = new PaymentOrderProcessor(
$this->repositoryMock,
$this->notificationServiceMock,
$this->apiClientMock,
$this->loggerMock,
$this->fraudDetectorMock
);
$result = $processor->processPayment($paymentOrder);
$this->assertInstanceOf(PaymentOrderProcessingResult::class, $result);
$this->assertFalse($result->isSuccess());
$this->assertEquals("Payment marked as suspicious", $result->getMessage());
}
}
The code is almost correct!
I applied following very simple fixes and removed assumptions about parameters and methods that do not exists.
And the test passed:
Prompt 2:
In the second prompt, given that ChatGPT has already parsed class definition of PaymentOrderProcessor and has written almost fully correct test for one of the scenarios, I can proceed with iterative prompting.
I will only provide the body of the method that was generated by ChatGPT with my simple corrections applied.
I made some fixes. Write more tests accordingly to the fixed example below. Include success scenario
{method body}
Output 2:
Here are more tests, including the success scenario:{rest of the test methods}
Whole output:
public function testProcessPaymentSuccessful(): void
{
$paymentOrder = new PaymentOrder(1, 100);
$this->fraudDetectorMock
->expects($this->once())
->method('isSuspicious')
->with($paymentOrder)
->willReturn(false);
$paymentOrderApiResponse = new PaymentOrderApiResponse(true);
$this->apiClientMock
->expects($this->once())
->method('sendPaymentOrderRequest')
->with($paymentOrder)
->willReturn($paymentOrderApiResponse);
$this->repositoryMock
->expects($this->once())
->method('savePaymentOrderResult')
->with($paymentOrder, $paymentOrderApiResponse);
$this->notificationServiceMock
->expects($this->once())
->method('sendNotification')
->with($paymentOrder, true);
$processor = new PaymentOrderProcessor(
$this->repositoryMock,
$this->notificationServiceMock,
$this->apiClientMock,
$this->loggerMock,
$this->fraudDetectorMock
);
$result = $processor->processPayment($paymentOrder);
$this->assertInstanceOf(PaymentOrderProcessingResult::class, $result);
$this->assertTrue($result->isSuccessful());
}
public function testProcessPaymentPaymentOrderRequestFailure(): void
{
$paymentOrder = new PaymentOrder(1, 100);
$this->fraudDetectorMock
->expects($this->once())
->method('isSuspicious')
->with($paymentOrder)
->willReturn(false);
$paymentOrderApiResponse = new PaymentOrderApiResponse(false, 'Payment failed');
$this->apiClientMock
->expects($this->once())
->method('sendPaymentOrderRequest')
->with($paymentOrder)
->willReturn($paymentOrderApiResponse);
$this->expectException(PaymentOrderRequestFailure::class);
$this->expectExceptionMessage('Payment failed');
$processor = new PaymentOrderProcessor(
$this->repositoryMock,
$this->notificationServiceMock,
$this->apiClientMock,
$this->loggerMock,
$this->fraudDetectorMock
);
$processor->processPayment($paymentOrder);
}
public function testProcessPaymentPersistenceException(): void
{
$paymentOrder = new PaymentOrder(1, 100);
$this->fraudDetectorMock
->expects($this->once())
->method('isSuspicious')
->with($paymentOrder)
->willReturn(false);
$paymentOrderApiResponse = new PaymentOrderApiResponse(true);
$this->apiClientMock
->expects($this->once())
->method('sendPaymentOrderRequest')
->with($paymentOrder)
->willReturn($paymentOrderApiResponse);
$this->repositoryMock
->expects($this->once())
->method('savePaymentOrderResult')
->with($paymentOrder, $paymentOrderApiResponse)
->willThrowException(new PersistenceException('Failed to save payment order result'));
$this->loggerMock
->expects($this->once())
->method('critical');
$processor = new PaymentOrderProcessor(
$this->repositoryMock,
$this->notificationServiceMock,
$this->apiClientMock,
$this->loggerMock,
$this->fraudDetectorMock
);
$result = $processor->processPayment($paymentOrder);
$this->assertInstanceOf(PaymentOrderProcessingResult::class, $result);
$this->assertFalse($result->isSuccessful());
}
Those tests are almost correct!
ChatGPT did not know about the second parameter $message of the PaymentOrderApiResponse class because it is never accessed in the class being tested.
I applied following fixes:
And now all of the tests passed!
Try it out!
With a bit of practice, you will get the feeling of how this kind of approach works for your various tasks including testing or writing code but you can learn to use it to speed up writing your unit tests fairly quickly.
Happy testing!
Top comments (0)