A non-square matrix is a matrix that has a different number of rows and columns. In the modified double-guard Hill cipher algorithm, a non-square matrix is used as the private key matrix that plays a role in the message encryption and decryption process. Therefore, the determinant of the key matrix is needed to obtain the inverse of the key matrix. Mirko Radić defined the determinant of matrix Amxn, m<=n as the signed sum of the determinants of the mxm submatrices as many as C (n, m). Radić’s determinant can be used to determine the general formula for the determinant of certain non-square matrices. The purpose of this research is to find out the determinant of matrix R = [\matrix (1&0&0&...&0&0@0&1&0&...&0&0@0&a_1&a_2&...&a_i&0@0&0&0&...&0&1)], ai ∈ R, ∀i=1,2,...,n-2 where n>4, using Radić’s determinant and an example of its use. The result of this research are the following theorem. If a non-square matrix R = [\matrix (1&0&0&...&0&0@0&1&0&...&0&0@0&a_1&a_2&...&a_i&0@0&0&0&...&0&1)], ai ∈ R, ∀i=1,2,...,n-2 where n>4 then |R|= Σ (-1)i+1 ai , for n odd and Σ (-1)i ai, for n even where i=2 to n-2. The use of the theorem is shown in an example problem using the modified double-guard Hill cipher where matrix R is chosen as the private key matrix. Several conditions must be met by the matrix R to be selected as the key matrix, including all elements of matrix R being positive integers, |R|\neq 0 , and R invertible in modulo 128.