How To Convert Audio from 2D to 3D

#javascript #android

Immersive audio is becoming an increasingly important factor for enhancing user experience in the music, gaming, and audio/video editing fields. The spatial audio function is ideal for meetings, sports rehabilitation, and particularly for exhibitions, as it helps deliver a more immersive experience. For users who suffer from visual impairments, the function can serve as a helpful guide.

In this article, I am going to reuse the sample code on this GitHub repo .I will implement spatial audio function in my android app and delivers the 3D surround sound.

Development Practice

Preparations

Prepare the audio for 2D-to-3D conversion, which is better a MP3 file. If not, follow the instructions specified later to convert the format to MP3 first. If the audio is part of a video file, just extract the audio first by referring to the instructions described later.

1.Configure the Maven repository address in the project-level build.gradle file.

buildscript {
    repositories {
        google()
        jcenter()
        // Configure the Maven repository address for the HMS Core SDK.
        maven {url 'https://developer.huawei.com/repo/'}
    }
    dependencies {
        ...
        // Add the AppGallery Connect plugin configuration.
        classpath 'com.huawei.agconnect:agcp:1.4.2.300'
    }
}
allprojects {
    repositories {
        google()
        jcenter()
        // Configure the Maven repository address for the HMS Core SDK.
        maven {url 'https://developer.huawei.com/repo/'}
    }
}

Add the following configuration under the declaration in the file header:

apply plugin: 'com.huawei.agconnect'

2.Add the build dependency on the Audio Editor SDK in the app-level build.gradle file.

dependencies{
    implementation 'com.huawei.hms:audio-editor-ui:{version}'
}

3.Apply for the following permissions in the AndroidManifest.xml file:

<!-- Vibrate -->
<uses-permission android:name="android.permission.VIBRATE" />
<!-- Microphone -->
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<!-- Write into storage -->
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<!-- Read from storage -->
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<!-- Connect to the Internet -->
<uses-permission android:name="android.permission.INTERNET" />
<!-- Obtain the network status -->
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<!-- Obtain the changed network connectivity state -->
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />

Code Development

1.Create app's custom activity for selecting one or more audio files. Return their paths to the SDK.

// Return the audio file paths to the audio editing screen.
private void sendAudioToSdk() {
    // Set filePath to the obtained audio file path.
    String filePath = "/sdcard/AudioEdit/audio/music.aac";
    ArrayList<String> audioList = new ArrayList<>();
    audioList.add(filePath);
    // Return the path to the audio editing screen.
    Intent intent = new Intent();
    // Use HAEConstant.AUDIO_PATH_LIST provided by the SDK.
    intent.putExtra(HAEConstant.AUDIO_PATH_LIST, audioList);
    // Use HAEConstant.RESULT_CODE provided by the SDK as the result code.
    this.setResult(HAEConstant.RESULT_CODE, intent);
    finish();
}

2.Register the activity in the AndroidManifest.xml file as described in the following code. When you choose to import the selected audio files, the SDK will send an intent whose action value is com.huawei.hms.audioeditor.chooseaudio to jump to the activity.

<activity android:name="Activity "> 
<intent-filter> 
<action android:name="com.huawei.hms.audioeditor.chooseaudio"/> 
<category android:name="android.intent.category.DEFAULT"/> 
</intent-filter> 
</activity>

Launch the audio editing screen. When you tap Add audio, the SDK will automatically call the activity defined earlier. Then operations like editing and adding special effects can be performed on the audio. After such operations are complete, the edited audio can be exported.

HAEUIManager.getInstance().launchEditorActivity(this);

3.(Optional) Convert the file format to MP3.
Call transformAudioUseDefaultPath to convert the format and save the converted audio to the default directory.

// Convert the audio format.
HAEAudioExpansion.getInstance().transformAudioUseDefaultPath(context,inAudioPath, audioFormat, new OnTransformCallBack() {
    // Callback when the progress is received. The value ranges from 0 to 100.
    @Override
    public void onProgress(int progress) {
    }
    // Callback when the conversion fails.
    @Override
    public void onFail(int errorCode) {
    }
    // Callback when the conversion succeeds.
    @Override
    public void onSuccess(String outPutPath) {
    }
    // Callback when the conversion is canceled.
    @Override
    public void onCancel() {
    }
    });
// Cancel format conversion.
HAEAudioExpansion.getInstance().cancelTransformAudio();

Call transformAudio to convert audio and save the converted audio to a specified directory.

// Convert the audio format.
HAEAudioExpansion.getInstance().transformAudio(context,inAudioPath, outAudioPath, new OnTransformCallBack(){
    // Callback when the progress is received. The value ranges from 0 to 100.
    @Override
    public void onProgress(int progress) {
    }
    // Callback when the conversion fails.
    @Override
    public void onFail(int errorCode) {
    }
    // Callback when the conversion succeeds.
    @Override
    public void onSuccess(String outPutPath) {
    }
    // Callback when the conversion is canceled.
    @Override
    public void onCancel() {
    }
    });
// Cancel format conversion.
HAEAudioExpansion.getInstance().cancelTransformAudio();

(Optional) Call extractAudio to extract audio from a video to a specified directory.

// outAudioDir (optional): directory path for storing extracted audio.
// outAudioName (optional): name of extracted audio, which does not contain the file name extension.
HAEAudioExpansion.getInstance().extractAudio(context,inVideoPath,outAudioDir, outAudioName,new AudioExtractCallBack() {
    @Override
    public void onSuccess(String audioPath) {
    Log.d(TAG, "ExtractAudio onSuccess : " + audioPath);
    }
    @Override
    public void onProgress(int progress) {
    Log.d(TAG, "ExtractAudio onProgress : " + progress);
    }
    @Override
    public void onFail(int errCode) {
    Log.i(TAG, "ExtractAudio onFail : " + errCode);
    }
    @Override
    public void onCancel() {
    Log.d(TAG, "ExtractAudio onCancel.");
    }
    });
// Cancel audio extraction.
HAEAudioExpansion.getInstance().cancelExtractAudio();

Call getInstruments and startSeparationTasks for audio source separation.

// Obtain the accompaniment ID using getInstruments and pass the ID to startSeparationTasks.
HAEAudioSeparationFile haeAudioSeparationFile = new HAEAudioSeparationFile();
haeAudioSeparationFile.getInstruments(new SeparationCloudCallBack<List<SeparationBean>>() {
    @Override
public void onFinish(List<SeparationBean> response) {
// Callback when the separation data is received. The data includes the accompaniment ID.
}
    @Override
    public void onError(int errorCode) {
        // Callback when the separation fails.
}
});
// Set the parameter for accompaniment separation.
List instruments = new ArrayList<>();
instruments.add("accompaniment ID");
haeAudioSeparationFile.setInstruments(instruments);
// Start separating.
haeAudioSeparationFile.startSeparationTasks(inAudioPath, outAudioDir, outAudioName, new AudioSeparationCallBack() {
    @Override
    public void onResult(SeparationBean separationBean) { }
    @Override
    public void onFinish(List<SeparationBean> separationBeans) {}
    @Override
    public void onFail(int errorCode) {}
    @Override
    public void onCancel() {}
});
// Cancel separating.
haeAudioSeparationFile.cancel();

Call applyAudioFile to apply spatial audio.

// Apply spatial audio.
// Fixed position mode.
HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.POSITION);
haeSpaceRenderFile.setSpacePositionParams(
                            new SpaceRenderPositionParams(x, y, z));
// Dynamic rendering mode.
HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.ROTATION);
haeSpaceRenderFile.setRotationParams( new SpaceRenderRotationParams(
                                    x, y, z, surroundTime, surroundDirection));
// Extension.
HAESpaceRenderFile haeSpaceRenderFile = new HAESpaceRenderFile(SpaceRenderMode.EXTENSION);
haeSpaceRenderFile.setExtensionParams(new SpaceRenderExtensionParams(radiusVal, angledVal));
// Call the API.
haeSpaceRenderFile.applyAudioFile(inAudioPath, outAudioDir, outAudioName, callBack);
// Cancel applying spatial audio.
haeSpaceRenderFile.cancel();