A matrix is a two-dimensional, rectangular or square array of numbers or functions. A determinant is always a square block of numbers or functions and is used in solving systems of simultaneous equations and is really an alternative way of representing a number (a scalar) or a polynomial and is evaluated following a single rule. A matrix, which may not necessarily be square, is a way of representing and handling linear transformations, which means, for example, if we want to rotate a picture of a protein or other molecule, a rotation matrix is used to take us from one orientation to another. The transformation is linear as each atom is moved (we could also say mapped) to a new position, but the molecular structure remains the same because the distance and angle between any two atoms is unchanged.
Matrices and determinants are used when calculating electronic energies, such as by the Huckel method, and in quantum mechanics generally. Expectation values, e.g. average energies, positions, momenta, etc., which are the quantities measured in an experiment, are, in quantum mechanics, also called matrix elements. When group theory is used to calculate the normal vibrational modes of a molecule or even the symmetry of a molecular orbital, matrices are involved.