To clarify your question - I assume you mean that if you get 4 or more A's in a row it's counted as a fail, and if you get two groups of 3 A's each separated by at least one B it's a success.
Start by considering how the three A's are arranged relative to B's:
Case 1. You can have the three A's as the first three letters followed by a B, and you don't care how the remaining 8 letters are arranged:: AAABxxxxxxxx.
Case 2. The first N letters can be anything, followed by the pattern BAAAB, and then the remaining 7-N letters can be anything; for example if N=2: xxBAAABxxxxx. This is valid for N=0 to N=7.
Case 3. The first 8 letters can be anything, followed by BAAA for the last four letters: xxxxxxxxBAAA.
For case 1 the number of ways that the last 8 letters can be arranged, given that they consist of 3 A's and 5 B's is:
For case 2 the number of ways that the remaining 7 letters can be arranged, given that there are 3 A's and 4 B's, is:
Note that case 2 can occur in 8 different ways, since the starting B can be in position 1, 2, 3, 4, 5, 6, 7, or 8. Thus the total number of ways that case 2 can occur is 8 x 35 = 280.
Finally for case 3 the math is the same as for case 1, so there are 56 ways it can occur.
Thus the total number of arrangements that yield 3 A's in a row, given that you have 6 A's and 6 B's, is 2x56+280 = 392.